arXiv:l502.03300v 1 [math.ST] 11 Feb 2015 


A sequential rejection testing method for 
high-dimensional regression with correlated variables 

Jacopo Mandozzi and Peter Biihlmann 
Seminar for Statistics, ETH Zurich 

February 12, 2015 


Abstract 

We propose a general, modular method for significance testing of groups (or clusters) of 
variables in a high-dimensional linear model. In presence of high correlations among the co¬ 
variables, due to serious problems of identifiability, it is indispensable to focus on detecting 
groups of variables rather than singletons. We propose an inference method which allows to 
build in hierarchical structures. It relies on repeated sample splitting and sequential rejection, 
and we prove that it asymptotically controls the familywise error rate. It can be implemented 
on any collection of clusters and leads to improved power in comparison to more standard non¬ 
sequential rejection methods. We complete the theoretical analysis with empirical results for 
simulated and real data. 

Keywords and phrases: Familywise error rate; Hierarchical clustering; High-dimensional 
variable selection; Inheritance procedure; Lasso; Linear model; Minimal true detection; Multiple 
testing; Sample splitting; Sequential rejection principle; Singleton true detection. 


1 Introduction 


Error control of false selection or false positive statements based on p-values is a primary goal of 
statistical inference and an established, broadly used tool in many areas of science. It relies on 
standard statistical hypothesis testing and procedures which give provable guarantees in presence 


of multiple, potentially very large scale multiple testing (Westfall, 1993; Dudoit and van der Laan 


2007, Efron, 2010). While being standard in the classical low-dimensional setup, statistical signif¬ 


icance testing in the more challenging high-dimensional setting where the number of variables p 
might be much larger than the sample size n has only received attention recently. 

We consider here a linear regression model 


Y = X/3° + e, e~AT n (0,a 2 /), 


( 1 ) 


with n x p design matrix X, p x 1 regression vector /3° and n x 1 response Y. We allow for high¬ 
dimensional scenarios where p> n. We assume that the regression coefficient vector is sparse with 
many coefficients of f3° being equal to zero, that is, the active set of variables 


s 0 = {j;$?o} 

is assumed to be a small subset of {1,... ,p} corresponding to all variables. 


1 












A few methods for assigning p-values and constructing confidence intervals for individual pa- 


rameters /3° (j = 1 ,,p) have been suggested ( 

Wasserman and Roeder 

2009 Meinshausen et al. 

2009} Biihlmann, 2013; Zhang and Zhang, 201-4 

l; van de Geer et ai.( 20 

14; Lockhart et al.|, 2014 

Javanmard and Montanan) 2014), and some of them have been cornpa 

red against each other in 
itial statements can easily 

various settings (Biihlmann et al. 2014; Dezeure et al. 2014). The inferei 


be adjusted for multiplicity, thanks to the methodology and theory in multiple testing (Dudoit and 
cf.). However, and important for practical applications, some major issues in 


van der Laan, 2007 


presence of highly correlated variables still need further attention: typically, when p>n, none or 
only a few of the individual /3^’s turn out to be significant which is a consequence of their near 
non-identifiability (even when some theoretical conditions on well-posedness on the design matrix 
X (Biihlmann and van de Geer, 2011 cf.) hold). However, a group of (correlated) variables is often 


much better identifiable, but one can then not determine anymore the relevant variables within 


such a group (Biihlmann et al., 2013; Meinshausen 2014; Mandozzi and Biihlmann 2015). 


Thus, our main goal is testing of significance of groups of parameters: for a group or cluster 
C C {1,... ,p} we consider the following null- and alternative hypothesis, respectively: 


H, 


0 ,c 


$ = 0 for all j E C, Ha,c ■ /?? / 0 for at least one j E C. 


Given a collection C of clusters, we propose a general method for obtaining a collection 1Z C C 
of rejected clusters such that familywise error rate (FWER) is strongly controlled. That is, for a 
given nominal level a E (0,1): 

F[TZ CJ]>l-a, 

where T = {C E C s.t. i7o,C is false} i.e., T is the collection of false null hypotheses. Our new 
method has the following main features: 


• It can be implemented on any collection of clusters C. 

• It is modular in the sense that it requires four basic building blocks that have to satisfy certain 
assumptions. 

• Its modular conception allows for a better insight of the procedure’s power and improvements 
thereof. 


We are particularly interested to use the procedure for hierarchically ordered clusters of (cor¬ 
related) variables. Such a hierarchical structure can be obtained from the output of a hierarchical 
clustering algorithm: since it operates on the design matrix X only and does not involve the re¬ 
sponses Y, the inference for j3° remains correct (for fixed design or by conditioning on X). With 
such a hierarchical cluster tree, our inference method (Sections 2.5 and 4.2) first tests the cluster 
C = {1,... ,p} containing all the variables (the top node in the tree): if the corresponding null- 
hypothesis is rejected, we test some refined clusters, and we proceed down the cluster tree, in a 
sequential manner, until a cluster is not significant anymore. Figures [l] and [2] in Section [5] provide 
some graphical illustrations. This procedure has the remarkable property that the resolution level 
of the significant clusters is automatically controlled by the sequential testing method: if the signal 
is strong (e.g. large absolute values of components of /3°) and the variables are not too highly 
correlated, one can detect small clusters or even single variables and vice-versa, if the signal isn’t 
very strong or the variables are highly correlated, only larger groups can be detected as significant. 
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Relation to other work. Our proposed method is based on the multi sample splitting method 
from Meinshausen et al. (2009) and the sequential rejection principle of Goeman and Solari (2010). 
It is a generalization and power improvement over the multi sample splitting technique for inference 


of single variables (Meinshausen et al. 2009) and for hierarchically ordered clusters of variables 


(Mandozzi and Biihlmann 2015). The improvement in power is strict, and in analogy to the 


gain of power of Holm’s procedure (Holm, 1979) over the Bonferroni adjustment. Thus, even if the 
increased power might be only small for some datasets, one cannot do worse with the new procedure. 
The only price to pay is a slightly more complicated algorithm: we provide an implementation in 
the R-package hdi. 


Outline of the paper. In Section[2]we describe the four basic building blocks of the method and 
the assumptions that are sufficient to establish in Section [3] its strong FWER control. In Sections 


singletons and hierarchically ordered clusters. In Section [4.3| we show how logical relationships can 
be used to improve the power. Finally, we provide in Section [5] a comparison based on empirical 
results for error control and power, with a focus on minimal true detections, and we apply the new 
method to a real dataset. 


4.1 and |4.2[ respectively, we focus on the inference of two specific kinds of cluster collections: 


2 A construction based on four building blocks 

Our method is based on four basic building blocks that satisfy certain assumptions. 

One main ingredient is multi sample splitting. For b = 1 where B is the number of 

repeated sample splitting, the original data of sample size n is split into two disjoint groups, 
and N^ t , i.e., a partition 

n} = N£ ) UN® 

is randomly chosen. The groups are chosen of equal size if n is even or satisfy | j¥^| | = || + 1 
if n is odd. 

The idea is to use data from to select a few variables and the other data from N^ t 
to perform the statistical hypothesis testing in the low-dimensional submodel with the selected 
variables from . The details are described next. 


2.1 Screening of variables 

We consider variable screening where an estimator C {!,... ,p}, based on data cor resp onding 


(*>) 


t° An 


is aiming at including all active variables Sq. A prime example is the Lasso (Tibshirani 


1996), while a detailed empirical comparison of five popular screening procedures can be founc 
in (Biihlmann and Mandozzi 2014). Assume that the screening procedure satisfies the following 


properties for any sample split b: 


(Al) Sparsity property: IS'®) < n/2. 

(A2) 5-Screening property: D So] > 1 — <5, where 0 < <5 < 1. 

The sparsity property in (Al) implies that for each sample split b it holds that < | N ^ t |, a 

condition which is necessary for applying classical tests as described in Section [2.2| below. The 5- 
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screening property in (A2) ensures that all the relevant variables are retained with high probability 
(where 5 > 0 is typically small). 

We indicate in Section [3d] that under some assumptions, the Lasso satisfies (Al) and (A2). 


2.2 Testing and p-values 


The idea is to perform a classical statistical test on the other half sample from N^j t in a low¬ 
dimensional problem with variables from only. 

For each sample split b, based on the second half of the sample corresponding to N^ t , consider 
a testing procedure, e.g. the classical partial F-test (see also Section 3.1), that provides correct 


p-values p C '^ for the null hypothesis H Q Cn §(b) for each screened set in the sense that for each 
nominal level a G (0,1) 


(A3) Correct testing property: Under the null hypothesis H f] Cn §(b) it holds P \p C ’^ < a\ < a. 


We note that the probability is with respect to the data generating random variables corresponding 
to the second half N^ t , and the null-hypothesis is fixed with respect to N^ t . Due to the screening 
property (A2), when <5 —>■ 0, the null-hypothesis H Q Cn §( b \ approximates the unconditional hypoth¬ 
esis Hq c which we aim to test for. If C D = 0 define p c V>) = \ This provides a (correct) 
p-value p c V) for each cluster C € C and each sample split b G {1... B}. 


2.3 Multiplicity adjustment 

Consider for each sample split b and each cluster C G C a multiplicity adjustment procedure 
nriQ : 2 C —> [l,oo] that for each collection TZ of rejected clusters provides a multiplicity adjustment 
rriQ {TZ) > 1 and satisfies the following properties: 


(A4) Monotonicity property: If 7 Z C S then m^) {JZ) > rn^{S). 

i{Cns® ^ 0 } 


(A5) Single-step property: 

c&c\n 


m 




< i. 


where we define l/oo = 0. Such a family of multiplicity adjustments for b = 1,... ,B are often 
naturally induced from a global multiplicity adjustment procedure me- 


2.4 Aggregation of p-values 

Consider a collection of screened sets of variables S^ b \ a cluster C € C, a collection of p-values 
pC,( h ) for the null-hypothesis Hq cnS^ (which approximates Fto.C; see comment after (A3)) and a 

collection of multiplicity adjustments m^‘) > 1 (we drop here the dependence on TV). 

The goal is to aggregate the p-values p c V)^ ; pC,( B ) f Q a single p-value which is adjusted for 

multiplicity. An aggregation procedure is a monotone increasing function aggr : [0, 1] B —> [0,1]. 
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Assume it satisfies the following property: 


(A6) Aggregation property: 


If P[p c >( 6 ) < a] < a, VaG [0,1], then 
Pfaggr^^’^m^P,... ,p c ^ B " > m^) < a\ 

< = 2^-7m-- V « G [°> !]• 


B 


6=1 


m, 


(b) 

c 


2.5 The procedure 

Our procedure is based on the four building blocks above. First , we proceed wi th sc reening of 
the variables based on the first half sample from (Section 2.1), e.g., in Section 


5.1 


Lasso with regularization parameter chosen by 10-fold cross-validation (see also Section 3.1). Then, 


we construct the p-values based on the second half sample from N$® t by using the partial F-test 


we use the 


2.2 


and see also Section 


3.1). This leads to a (correct) p-value p c ^ h l for each cluster C £ C 


(Section 

and each sample split b £ {1... B}. 

The multiplicity adjustment is done sequentially (Section |2.3[ ). Based on a chosen significance 
level a € (0,1) and for a collection of currently rejected sets 7Z, define the successor of 7Z as 

= {C £ C\1Z s.t. aggr(p c '’ < ' 1 ' ) m^(7 ?.),...,p c < a} 

Start from “no rejections” TZq = 0, define TZ l+ \ = lZiUAT(lZi) and TZoc = lim^oo IZj (although TZoc 
is never constructed due to finite-ness of all possible subset of the variables). Concrete choices of 
{TZ),... ,m^\lZ) are discussed in Section 


Finally, we aggregate the p-values as indicated in Section |2.4 
are described in Proposition [l] in Section 3.1 


Concrete aggregation methods 


3 Familywise error control 


We show here that the method from Section 2.5 (strongly) controls the FWER at each step i = 

0 , 1 , 2 ,... 


Theorem 1. Assume that (A1)-(A6) hold. Then for any i e No U oo 

P^CJ] > (1 -5) B -a, 

where T = {C € C s.t. C n Sq / 0} is the collection of false null hypotheses. 
A proof is given in the Appendix. 


3.1 Screening, testing and aggregation: their properties 

We discuss here some choices for screening, testing and aggregation which we use in the implemen¬ 
tation in the R-package hdi. The issue of sequential multiplicity adjustment is treated separately 
in Section 0J 

For variable screening, we use the Lasso with regularization parameter chosen by 10-fold cross- 
validation. Theoretical justification of the sparsity and screening property (Al) and (A2) can be 
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derived by assuming a compatibility or restricted eigenvalue condition on the fixed design matrix 
X and a beta-min assumption requiring that min je 5 0 |/3?| 3> ^/|So| log(p)/n is sufficiently large: 
we refer to Biihlmann and van de Geer (2011 Ch. 2.7 and Ch. 6 ) for the details. 

For construction of the p-values (in the low-dimensional setting, due to variable screening in 
the first half of the sample) we use the partial F-test. Then, assuming fixed design X and Gaussian 
errors, condition (A3) holds. 

For aggregation of the p-values, ensuring that (A 6 ) holds, we have the following result for two 
slightly different methods. 

Proposition 1. Denote by q^(u) the empirical 'y-quantile of the values occurring in the components 
of a vector u. The monotone increasing functions [0, 1] B —> [0,1] 


(P { 1 \...,P (B) ) 


Q{l) — min { 1 , q- y (p (1) /'y,...,p {B) /'y)} 
min { 1 , (1 - log 7 min ) inf 

7S(7mm,l) 


Q( 7)} 


satisfy the aggregation property (A6) for any 7 , 7 m in £ (0,1). 


A proof, which was basically given in Meinshausen et al. (2009), can be found in the Appendix. 


4 Some concrete methods for multiplicity adjustment 

We discuss here the issue of multiplicity adjustment, and justify assumption (A4) and (A5) for 
different inference procedures. 


4.1 Inference of single variables 


This first example is paradigmatic for the advantages of the modular approach: a simple improve¬ 
ment of the multiplicity adjustment procedure allows for a better power, basically in the same way 


as in a low-dimensional setting in (Goeman and Solari, 2010). 


Concretely, we consider the problem of inferring single variables, i.e., the collection of clus¬ 
ters C = = 1 ,...,p}. The method proposed in Meinshausen et al. (2009) corresponds to 

the method of Theorem [I] with the aggregation procedures of Proposition [l] and the following 
Bonferroni-based (Bonferroni, 1936 Dunn, 


1961) multiplicity adjustment procedure: 


m 


(77) = |5 (b) |. 


(2) 


As the multiplicity adjustments are independent from the (previously) rejected collection of sets, 
the monotonicity property (A4) is trivially satisfied, while the single-step property (A5) follows 
from 


X 

CeC\Tl 


ljcnsw £ 0} 


X 

{i}ec \n 


\{{i} n 5 W £ 0} 

| 5 ( fe )| 


< 1 . 


The power of the method can be improved taking instead of © the following Bonferroni-Holm- 
based (Holm, 1979) multiplicity adjustment procedure: 


m 


(77) = I S® n (C \ 77)1 = \{j g S® s.t. {j} i 77}I. 


(3) 
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The monotonicity property (A4) is still satisfied since | n (C \ TZ)\ > | n (C \ <S)| for TZ C S, 
whereas 


E 

ce c\tz 


i{cns<v / 0 } 


m, 


(*)/ 


proves the single step property (A5). 


E 

{*}£ C\R 


!{{i} n S® £ 0} 

|n {c\n)\ 


= l 


4.2 Inference of hierarchically ordered clusters of variables 

When dealing with the challenge of inferring hierarchically ordered clusters of variables, e.g. from 
the tree-structured output of a hierarchical clustering algorithm, one considers a collection of clus¬ 
ters C = {Ci}i where for any two clusters C{. C# e C, either one cluster is a subset of the other, or 
they have an empty intersection. The method proposed in Mandozzi and Biihlmann (2015, Section 


2), which is based on the procedure of |Meinshausen| (2008), corresponds to the one as in Theorem 
[I] with the aggregation methods of Proposition [I] and the following multiplicity adjustment: 

oo, if anc(C) (2 1Z 

m { c\lZ) = -1^- 


IsWnCj' 

1 , 


if anc(C) C TZ and S (6) D C + 
otherwise. 


(4) 


Here, anc(C) denotes the ancestors in a hierarchically ordered cluster tree. To check the mono¬ 
tonicity property (A4), consider TZ C S. For C E C with anc(C) C 7£ it holds anc(C) C S and 

,„u 0 n a r ,„uv, r.(n\ rt t? 


hence (TZ) = m^ J (S), while for C 6 C with anc(C) % TZ one has (K) = oo > m^(S). The 


single step property (A5) follows from 
1 {C n S W / 0} 


E 

ce c\n 


m 


(TZ) 


| 5 ( fe )| 


E 


\s w nc\ < l, 


CeC\R s.t. anc(C) ctz 


where in the inequality we have used the fact that for two sets in the sum above, one cannot be a 
subset of the other and hence, by definition of the hierarchy C, they are disjoint. 


4.2.1 The inheritance procedure in the high-dimensional setting 


In Goeman and Solari (2010, Section 6.3) and Goeman and Finos (2012), the authors propose 
various possibilities on how the sequential rejection principle can be used to improve the power 
of the hierarchical procedure in Meinshausen (2008). We consider here the most powerful one, 


the inheritance procedure of Goeman and Finos (2012) which we extend to the high-dimensional 
setting with hierarchical cluster trees. In order to do that, we apply the method of Theorem [T] with 
the aggregation procedures of Proposition [T] and the following multiplicity adjustment: 


mf}j(TZ) = < 


oo, 

1 , 

|5W| 


Jwnc\ nDeanc(C) n D > otherwise, 


if anc(C) %. TZ 

if anc (C) C 7 Z and S ^ n C 


(5) 


where 


•S’- 


nh (TZ) = 


|5( b ) n D\ 


|S (6) n£| 


Ee ch(D)\£(K) 
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and 


£(K) = {C €C s.t. of(C') C Rj 


the set of extinct branches, i.e., the set of hypotheses which have been rejected together with all 
their offsprings denoted by of(C) (as before, anc(C) denotes the ancestors of cluster C ). Note that 
since rip (TZ) < 1 this procedure leads to an uniform improvement over the method of the previous 
section. 

The monotonicity property (A4) follows from the same considerations as above and 
TZ C 5 =*► £(TZ) C £(S) => n^iTZ) > n^ ] (S). 

To check that the single step property (A5) holds note that 


E 

ce c\n 


ljcns^ / 0} 


m 


(R) 


E 

Cec\n s.t. anc (C)cn 


15 ^ nc\ 


n 


\S^ n D\ 


Deanc(C) ^Eech(D)\£(-R.) 1^ n E \ 


E 

Ce c\n 


a 


(K) 


a 


where atg is as in 


Goernan and Finos 


(2012 


equation (5)) with the weights w^ n C |; 

therefore the single step property follows directly from the considerations in |Goeman and Finos| 

d2012b. 


4.3 Exploiting logical relationships: Shaffer improvements 


Logical relationships between hypothesis can be exploited to improve the power of the sequential 
rejection procedure. A first example of such an improvement for hierarchically ordered clusters 
was given in Meinshausen (2008), while in Goernan and Finos (2012) the improvement is applied 


to the inheritance procedure. Since those improvements are based on the considerations of Shaffer 
(1986) they are called “Shaffer improvements”. For the high-dimensional setting a possible Shaffer 
improvement consists of multiplying the multiplicity adjustment rri^) (7Z) with the Shaffer factor 

Sq\ 1Z) = max{m£?(W)/m^(7£) s.t. C Z^TZ, U congruent}, (6) 


where a set U C C is called congruent if, by the logical implications, it can be a complete set of 
false hypothesis (e.g. for a collection C of hierarchically ordered hypothesis U C C is congruent if 
for each C G U it holds anc (C) C U and at least one offspring leaf node of C is in U). 


Note that multiplication with the Shaffer factor never decreases the power of the method since 
by the monotonicity property (A4), Sq\TZ) < 1. Moreover s~q (7Z) = 1 if 1Z is congruent and since 
the collection T of all false hypothesis is congruent, the Shaffer improvement doesn’t affect the 
validity of equation Q. Finally, for TZ C S, 

ttlq (TZ)s^\tZ) = rnaxjm^ (U) s.t. C £ U D TZ, U congruent} 

> max{ra®(h) s.t. C £ U Z) S, U congruent} 

= m ( §{S)s ( §{S) 
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and hence the Shaffer improvement doesn’t affect the validity of equation 0 neither. 


We want to apply this Shaffer improvement to the inheritance procedure described in Section 
4.2. 1| Following the same reasoning as in Goeman and Finos (2012 Section 6 ), with the weights 


Wq= IS ^- 1 n C | we get the Shaffer factor 




#\n) = 


c 


+u' c —v t 


1 , 




c 


if c i n,si{c ) c c\n 

otherwise, 


where si(C) = ch{pa(C)}\{C} denotes the siblings of C, L C C denotes the collection of leaf nodes, 


u, 


C 


= £ 


Desi(C) W D and V C = min Desi(C) W D 

|5^>nc| 


sg } (^) = < \smnC\+\sWnsi(C)\ ] 


If C is a binary tree the Shaffer factor becomes 


if C i K,si(C) C£\H 
otherwise. 


Unlike as for the inheritance procedure in ([5]), the Shaffer factor ([ 6 ]) for the procedure in Q is 
always 1. Nevertheless, a possibility how to exploit logical relationships to improve the power of 
the procedure Q for binary trees, which provides a Shaffer improvement very similar to the one 
above, is illustrated in Mandozzi and Biihlmann (2015). 


5 Empirical results 

5.1 Implementation of the methods and considered scenarios 


In this section we compare the performance of the four methods illustrated in Sections |4.1| and 4.2 

i.e. single variable method with Bonferroni multiplicity adjustment ([2]), 


and refined in Section 4.3 


hierarchical method with Bonferroni-based adjustment Q along with Shaffer improvement as in 
Mandozzi and Biihlmann (2015), single variable method with Bonferroni-Holm multiplicity adjust- 
ment _ [3]) and hierarchical method with inheritance procedure ([5]) along with Shaffer improvement 
Q. In the following we refer to the first two methods as the “non-sequential methods” (strictly 
seen, the hierarchical method with Bonferroni-based adjustment is actually sequential, but there 
previous rejections are not used to improve subsequent multiplicity corrections) and the latter two 
methods as the “sequential methods”. 

We consider the same implementation of the methods and the same scenarios (with exactly the 


same sample splits) as in Mandozzi and Biihlmann (2015), although here we use only standard 
hierarchical clustering for the hierarchical methods. Concretely, the following choices have been 
made for implementation: 

• construction of the clusters with standard hierarchical clustering (using the R-function hclust) 
with distance between two covariables equal to 1 minus the absolute correlation between the 
covariables, and using complete linkage; 


screening with the Lasso (Tibshirani, 1996) with regularization parameter chosen by 10-fold 
cross-validation; 


B = 50 sample splits (for each scenario exactly the same splits as in Mandozzi and Biihlmann 


(2015)); 


9 


























for aggregation, the p-values in Proposition[l]are computed over a grid of 7 -values between 
7 min = 0.05 and 1 with grid-steps of size 0.025; 

nominal significance level a = 5%. 


The following scenarios are considered (for the details we refer to Mandozzi and Buhlmann (2015)): 

• 42 scenarios based on 7 designs; 

• for each design we consider 6 settings by varying the number of variables p in the model and 
the signal to noise ratio defined by SNR = \J (/3 0 ) r X 7 ’X/3°n _ 1 (j _2 , namely for p = 200 we 
use SNR = 4 and SNR = 8, for p = 500 we use SNR = 8 and SNR = 16 and for p = 1000 we 
use SNR = 16 and SNR = 32; 

• 3 designs based on synthetic data (“equi correlation”, “high correlation within small blocks” 
and “high correlation within large blocks”) and 4 designs based on semi-real data (“Riboflavin 
with normal correlation”, “Breast with normal correlation”, “Riboflavin with high correla¬ 
tion”, “Breast with high correlation”); 

• sparsity sq = 6 for the two “Riboflavin”-designs and sq = 10 for the other five designs. 


5.2 Familywise error rate control (FWER) 

For each of the 42 scenarios described in Section [5.1| we consider exactly the same 100 independent 


simulation runs as in Mandozzi and Buhlmann (2015, Section 4.2.2) by varying only the synthetic 
noise term e and count the number where at least one false selection is made. According to Theorem 
[IJ we expect this number to be at most 100 a = 5 (a = 0.05). The results for the Bonferroni-based 
methods can be seen in Mandozzi and Biihlmann (2015, Table 1): FWER control holds for 40 of 


the 42 scenarios and in 37 scenarios there is no false selection at all. 

The results for the methods with sequential rejection are very similar, the only differences being 
that for the “high correlation within small blocks”-design with p = 500 and SNR = 8 the number 
of runs with at least a false selection increases (compared to Bonferroni-type methods) from 7 to 
9 for the single variable method, and from 7 to 13 for the hierarchical method, respectively; for 
the same design with p = 1000 and SNR = 16 the number of runs with at least a false selection 
increases from 5 to 6 for both the single variable and hierarchical method. For all other scenarios, 
inclusively the “high correlation within large blocks”-design with p = 200 and SNR = 4, where 
the non-sequential hierarchical method slightly failed to control FWER (6 runs with at least a 
false detection), the sequential methods exhibit the same FWER control as their non-sequential 
counterparts. 

Summarizing, FWER holds for all four methods in 39 out of 42 scenarios and the designs where 
it doesn’t fully hold are “high correlation within small blocks” and “high correlation within large 
blocks”, which is not surprising since each active predictor is highly correlated with a false variable 
from and hence it is rather difficult for our screening method (the Lasso) to guarantee that 
SDS 0 . 


5.3 Power 

For measuring the power we consider four different aspects: the one-dimensional statistics defined 


in Mandozzi and Buhlmann (2015, Section 4.2.1) as “Performance 1” and “Performance 2” (see 
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below), the number of minimal true detections (MTDs, i.e., smallest significant groups of variables 
of any cardinality, containing at least one active variable, see below) and singleton true detections 
(STDs, i.e., MTDs with cardinality 1). Concretely, a cluster is said to be a MTD if it satisfies all 
of the following: 


• C is a significant cluster, e.g., has p-value < 5% (“Detection”); 

• There is no significant sub-cluster D C C (“Minimal”); 

• C £ To, i.e., there is at least one active variable in C (“True”); 
and we define: 


Performance 1 


Performance 2 


1 

isoi 


E 

MTD C 


1 


\cy 


i 

ISoI 


E 

MTD C with 


|C| <20 


2 MCI 



For each of the 42 scenarios outlined in Section 5.1 we consider exactly the same 100 inde¬ 


pendent simulation runs obtained in Mandozzi and Biihlmann (2015, Section 4.2.3-4) by varying 
the synthetic noise term e and the synthetic regression vector f3 . We then calculate the average 
Performance 1, Performance 2, number of MTDs and number STDs, over the 100 simulation runs. 
The results are shown in Table [l] for low SNR and Table [2] for high SNR (for the single variable 
methods each MTD is an STD and by definition Performance 2 is the same as Performance 1). 

Considering both low and high SNR, the methods with sequential rejection improve the con¬ 
sidered power measures in comparison to the analogous method without sequential rejection in 
207 out of 252 cases, the absolute improvement being at least 0.05 for MTDs and STDs, and at 
least 0.5 percent for Performance 1 and Performance 2 in 133 cases out of 252 cases. For better 
interpretation of these results: an absolute improvement of 0.05 MTDs (resp. STDs) basically 
means that in one out of 20 runs one more MTD (resp. STD) could be detected. Averaging over 
all scenarios, the improvement given by the sequential rejection procedures lies between 0.04 and 
0.06 for MTDs and STDs, and between 0.5 and 0.7 percent for Performance 1 and Performance 
2. The biggest gain with sequential rejection can be found in the “high correlation within small 
blocks”-design with p = 200 and low SNR: it consists of 0.48 more STDs, 0.33 more MTDs and an 
absolute increase of 3.8 percent of Performance 1 and 3.3 percent of Performance 2, respectively. 
This basically means that in half of the runs the method with sequential rejection could find one 
STD more and in one third of the runs it could find one MTD more. Other particularly favorable 
scenarios for an improvement with sequential rejection are the “equi correlation”-design and the 
“breast normal corr”-design, both with p = 200 and low SNR and the “high correlation within 
small blocks”-design with high SNR and p = 200, resp. p = 500. 

In general, the improvement given by the sequential rejection procedures decreases with in¬ 
creasing number p of covariables and is substantial only when the power of the method without 
sequential rejection is intermediate. These empirical findings are not surprising, since looking at 
how the methods are defined and in particular at the equations and 0, we conclude 

that an improvement with the sequential rejection methods is only possible if the related non¬ 
sequential method provides at least an STD (and gets more likely the more STDs are provided by 
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Design 

P 

low SNR 

# MTDs 

# STDs 

Perf 1 

Perf 2 

SB 

SH 

HB 

HSR 

HB 

HSR 

SB 

SH 

HB 

HSR 

HB 

HSR 

equi 

corr 

200 

4.79 

5.00 

5.40 

5.59 

4.34 

4.55 

47.9 

50.0 

44.2 

46.3 

46.1 

48.1 

500 

3.97 

4.13 

4.74 

4.84 

3.73 

3.84 

39.7 

41.3 

37.7 

38.7 

38.3 

39.3 

1000 

1.77 

1.79 

2.54 

2.54 

1.73 

1.73 

17.7 

17.9 

17.4 

17.4 

17.6 

17.6 

small 

blocks 

200 

4.45 

4.78 

6.85 

7.12 

4.36 

4.84 

44.5 

47.8 

53.7 

57.5 

60.7 

64.0 

500 

3.15 

3.33 

5.18 

5.27 

3.15 

3.42 

31.5 

33.3 

38.3 

40.1 

44.2 

45.5 

1000 

1.31 

1.35 

2.53 

2.57 

1.31 

1.37 

13.1 

13.5 

15.1 

15.6 

17.2 

17.7 

large 

blocks 

200 

0.29 

0.30 

6.50 

6.50 

0.28 

0.28 

2.9 

3.0 

6.7 

6.7 

31.3 

31.3 

500 

0.06 

0.06 

2.76 

2.76 

0.06 

0.06 

0.6 

0.6 

1.1 

1.1 

1.1 

1.1 

1000 

0.00 

0.00 

0.60 

0.60 

0.00 

0.00 

0.0 

0.0 

0.1 

0.1 

0.1 

0.1 

Riboflavin 

normal 

corr 

200 

1.41 

1.43 

2.41 

2.46 

1.33 

1.35 

23.5 

23.8 

23.4 

23.8 

25.2 

25.7 

500 

0.90 

0.90 

1.84 

1.85 

0.77 

0.79 

15.0 

15.0 

13.5 

13.8 

14.2 

14.5 

1000 

0.72 

0.73 

1.60 

1.63 

0.63 

0.66 

12.0 

12.2 

10.8 

11.2 

11.0 

11.4 

Breast 

normal 

corr 

200 

4.05 

4.16 

5.00 

5.11 

3.84 

3.94 

40.5 

41.6 

39.5 

40.6 

41.6 

42.9 

500 

3.95 

4.02 

5.04 

5.11 

3.82 

3.87 

39.5 

40.2 

38.8 

39.3 

39.6 

40.2 

1000 

3.30 

3.34 

4.25 

4.27 

3.10 

3.13 

33.0 

33.4 

31.2 

31.5 

31.7 

31.9 

Riboflavin 

high 

corr 

200 

1.44 

1.49 

2.96 

2.96 

1.41 

1.44 

24.0 

24.8 

26.0 

26.4 

31.8 

32.1 

500 

1.72 

1.79 

2.95 

2.98 

1.69 

1.72 

28.7 

29.8 

29.9 

30.4 

32.8 

33.3 

1000 

1.51 

1.51 

2.54 

2.56 

1.49 

1.52 

25.2 

25.2 

25.3 

25.8 

25.7 

26.1 

Breast 

high 

corr 

200 

3.98 

4.10 

5.91 

5.95 

3.87 

3.91 

39.8 

41.0 

41.2 

41.6 

46.1 

46.6 

500 

5.13 

5.22 

6.51 

6.56 

4.87 

4.93 

51.3 

52.2 

49.9 

50.4 

51.7 

52.3 

1000 

4.73 

4.77 

5.95 

5.98 

4.64 

4.67 

47.3 

47.7 

47.0 

47.3 

48.3 

48.6 

Average 

2.51 

2.58 

4.00 

4.06 

2.40 

2.48 

27.5 

28.3 

28.1 

28.8 

31.3 

31.9 


Table 1: Number of MTDs, number of STDs, Performance 1 in % and Performance 2 in %, averaged 
over 100 simulation runs, for single variable method with Bonferroni (SB), single variable method 
with Bonferroni-Holm (SH), hierarchical method with Bonferroni (HB) and hierarchical method 
with sequential rejection induced by the inheritance procedure (HSR). Scenarios with low SNR. 


the non-sequential method). Moreover, an improvement with sequential rejection is more likely to 
happen when the number IS) of screened variables is small. 

For a better illustration of what kind of an improvement is possible using sequential rejection, 
we show in Figures [T] and [2] the dendrograms (in gray) for a paradigmatic simulation run of the 
“equi correlation”- and the “high correlation within small blocks”-design, respectively, both with 
p = 200 and SNR = 4. Figure |T] illustrates that sequential rejection allows the detection of a further 
singleton, increasing the number of STDs from 6 to 7 and the number of MTDs 8 to 9. In Figure 
[2] sequential rejection allows to detect a singleton that could previously only be detected together 
with another non-relevant variable in a cluster of cardinality 2, increasing the number of true STDs 
from 4 to 5 (while the number of MTDs remains to be 6). 

Finally, we have performed a simulation with the same scenarios (and the same sample splits) 
as in Mandozzi and Biihlmann (2015, Section 4.3), i.e. “small blocks”-designs and “large blocks”- 
designs with 8 different correlations p G {0,0.4,0.7,0.8,0.85,0.9,0.95,0.99}. The full results are 
shown in Tables [3] and [4] in the Appendix. While the methods with sequential rejection control the 
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Design 

P 

high SNR 

# MTDs 

# STDs 

Perf 1 

Perf 2 

SB 

SH 

HB 

HSR 

HB 

HSR 

SB 

SH 

HB 

HSR 

HB 

HSR 

equi 

corr 

200 

9.77 

9.83 

9.79 

9.80 

9.73 

9.74 

97.7 

98.3 

97.4 

97.5 

97.4 

97.5 

500 

7.28 

7.38 

7.63 

7.67 

7.18 

7.24 

72.8 

73.8 

72.0 

72.5 

72.1 

72.6 

1000 

2.81 

2.84 

3.50 

3.50 

2.78 

2.78 

28.1 

28.4 

27.9 

27.9 

28.1 

28.1 

small 

blocks 

200 

9.18 

9.31 

9.98 

10.00 

9.29 

9.48 

91.8 

93.1 

96.3 

97.4 

98.1 

98.7 

500 

6.99 

7.03 

8.05 

8.14 

7.02 

7.15 

69.9 

70.3 

73.5 

74.7 

76.4 

77.4 

1000 

2.26 

2.27 

3.40 

3.41 

2.26 

2.28 

22.6 

22.7 

24.3 

24.5 

26.3 

26.5 

large 

blocks 

200 

2.17 

2.26 

9.58 

9.58 

2.13 

2.14 

21.7 

22.6 

27.9 

28.0 

61.4 

61.4 

500 

1.17 

1.20 

5.38 

5.38 

1.15 

1.15 

11.7 

12.0 

12.6 

12.6 

13.2 

13.2 

1000 

0.43 

0.45 

1.11 

1.11 

0.43 

0.43 

4.3 

4.5 

4.4 

4.4 

4.4 

4.4 

Riboflavin 

normal 

corr 

200 

3.39 

3.46 

3.89 

3.92 

3.33 

3.34 

56.5 

57.7 

56.3 

56.5 

58.7 

59.1 

500 

2.24 

2.25 

2.90 

2.90 

2.15 

2.15 

37.3 

37.5 

36.4 

36.4 

36.9 

36.9 

1000 

0.98 

1.00 

1.83 

1.83 

0.96 

0.96 

16.3 

16.7 

16.2 

16.2 

16.3 

16.3 

Breast 

normal 

corr 

200 

8.65 

8.70 

8.89 

8.93 

8.60 

8.65 

86.5 

87.0 

86.4 

86.8 

87.2 

87.7 

500 

6.81 

6.86 

7.33 

7.35 

6.72 

6.74 

68.1 

68.6 

67.6 

67.8 

68.3 

68.5 

1000 

3.95 

3.97 

4.81 

4.84 

3.79 

3.82 

39.5 

39.7 

38.1 

38.4 

38.4 

38.7 

Riboflavin 

high 

corr 

200 

3.86 

3.97 

4.79 

4.83 

3.82 

3.86 

64.3 

66.2 

66.0 

66.7 

69.4 

70.1 

500 

3.69 

3.72 

4.40 

4.43 

3.65 

3.68 

61.5 

62.0 

61.8 

62.3 

63.7 

64.2 

1000 

2.48 

2.51 

3.24 

3.27 

2.43 

2.45 

41.3 

41.8 

40.7 

41.0 

41.1 

41.4 

Breast 

high 

corr 

200 

9.09 

9.15 

9.59 

9.61 

9.09 

9.14 

90.9 

91.5 

91.9 

92.3 

93.8 

94.1 

500 

7.75 

7.82 

8.38 

8.40 

7.71 

7.72 

77.5 

78.2 

77.8 

78.0 

78.9 

79.1 

1000 

5.85 

5.89 

6.73 

6.76 

5.72 

5.75 

58.5 

58.9 

57.5 

57.9 

58.1 

58.5 

Average 

4.80 

4.85 

5.96 

5.98 

4.76 

4.79 

53.3 

53.9 

54.0 

54.3 

56.6 

56.9 


Table 2: Number of MTDs, number of STDs, Performance 1 in % and Performance 2 in %, averaged 
over 100 simulation runs, for single variable method with Bonferroni (SB), single variable method 
with Bonferroni-Holm (SH), hierarchical method with Bonferroni (HB) and hierarchical method 
with sequential rejection induced by the inheritance procedure (HSR). Scenarios with high SNR. 


FWER in exactly the same scenarios where it is also controlled by the non-sequential methods, they 
increase the average number of MTDs from 5.51 to 5.62 for the single variable method, and from 
8.11 to 8.18 for the hierarchical method, and the number of STDs for the hierarchical method from 
5.44 to 5.55, with improvements for a single scenario up to 0.48 MTDs and 0.56 STDs (averaged 
over 100 runs). 

The empirical results can be summarized as follows. The methods with sequential rejection 
essentially controls the FWER in the same way as the non-sequential methods. Regarding power, 
sequential rejection allows for improvements, to a similar extent for the single variable and the 
hierarchical procedures. As already noted in Mandozzi and Biihlmann (2015), for the non-sequential 
methods, the hierarchical methods have similar STDs as the single variable methods but allow for 
substantially more MTDs. Thus, our proposed hierarchical method with the inheritance procedure 
can be seen as the best of the considered methods. 
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hierarchical method with Bonferroni multiplicity adjustment 




Figure 1: Dendrograms for a paradigmatic simulation run of the “equi correlation”-design with 
p = 200 and SNR = 4. The active variables are labeled in black and the truly detected non-zero 
variables along the hierarchy are depicted in black. 


5.4 Real data application: Motif Regression 


We consider here a problem of motif regression (Conlon et al. 2003) from computational biology. 


We apply the four methods described above, plus the two hierarchical methods (with and without 


sequential rejection) using the recently proposed canonical correlation clustering of of Biihlmann 


et al. (2013), to a real dataset with n = 287 and p = 195, used in Meinshausen (2008, Section 


4.3) and Mandozzi and Biihlmann (2015, Section 4.4). The sequential rejection methods detects 
exactly the same significant structures as non-sequential methods, namely a single variable and a 
cluster containing 165 variables (the latter can be detected only with the hierarchical method with 
canonical correlation clustering). This can barely be considered as surprising, as with only one 
STD by the non-sequential methods, further improvements by the sequential methods are rather 
unlikely (see Section 5.3 for more explanation and empirical evidence). 


6 Conclusions 

We propose a general sequential rejection testing method for clusters and single variables in a 
high-dimensional linear model. In presence of high correlations among the covariables, due to 
serious problems of identifiability, it is essentially mandatory to focus on detecting significant 
groups of variables rather than single individual covariates. Our method asymptotically controls 
the familywise error rate (FWER), while, as a consequence of its modular structure, allowing for 
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hierarchical method with Bonferroni multiplicity adjustment 



hierarchical method with inheritance procedure 



Figure 2: Dendrograms for a paradigmatic simulation run of the “high correlation within small 
blocks”-design with p = 200 and SNR = 4. The active variables are labeled in black and the truly 
detected non-zero variables along the hierarchy are depicted in black. 


unburdened power optimization. We provide an implementation in the R-package hdi. 

We use and study the procedure for inference of single variables but much more importantly, 
for hierarchically ordered clusters of variables. With the latter, we establish a powerful scheme 
for meaningful inference in a high-dimensional regression model, much beyond considering single 
variables only. Our presented mathematical analysis on control of the FWER and power improve¬ 
ment is complemented by empirical results based on semi-real and simulated data confirming the 
theoretical results. 


7 Appendix 

7.1 Proof of Theorem [Q 


Proof. We show that the procedure satisfies monotonicity and single-step conditions as required by 
Goeman and Solari (2010, Theorem 1), i.e. 


77 C 5 ^ AT(77) C A f(S) U 5 (7) 

F\Af(T) CJ]>(1- 5) b - a. (8) 


Assume 77 C S and C G A/’(77). Then by definition &ggr{p C '^m^) (fR), ..., p c '4 s ) rn^\lZ)) < a. 
The monotonicity property (A4) of the multiplicity adjustment and the fact that the aggregation 
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procedure is monotone increasing imply 


aggr(p c, W... ,p C ’^m ( ' B \s)) < aggr (p c, ^m^ (TZ), ... ,p C ' < ' B ^m' B \ 77)) 
and hence either C G S or C G A f(S) which proves ([?]). Consider the event 

A = {S (b) DS 0 yb = 1...B} 

where all screenings are satisfied. Because of the ^-screening assumption (A2) it holds 
P{A) > (1 — 5) B and hence 

¥[M(P) = F[Af(P) %P\A] P(^) + F[Af(F) %P\A C ] P(7l c ) 

< F[M{P) % P\A) + (!-(! 


Since 


F[Af(T) %T\ A] 


< 


< 


p [lJ {^^{p C ’ W rnc\j r ),...,p C ^ B) rn^ ) (J 7 )) < a}] 


C\T 


X] p [ a gg AP C ' W mc\p), ■ ■ ■ ,P C ’ <<B) rn ( 'c\j r )) < a}] 
C\F 


(A3)(A6) 

< 


(A5) 

< 


^ a A l{CnS® ^0 } _ a 

C\F 6=1 


™$\r) 


™ i{cns® + o} 


6=1 C\F 


a 

B 


B 


J ] 1 <a 


6=1 


we conclude PfA/X-P") C p] = 1 - PfA/X-T 7 ) % P] > 1 — (a + (1 — (1 — 5 ) B )) = (1 — 5) B — a which 
proves (| 8 |) . □ 


7.2 Proof of Proposition [T] 


Proof. The proof was basically given in the Appendix of Meinshausen et al. (2009). 

In the following we omit the function min{l, •} from the definition of Q( 7 ) in order to simplify the 
notation (this is possible since the level a is smaller than 1). Define for u G (0,1) the function 


*■(«) : = Jj 1 tP (h> - u }■ 

D 6=1 

Then it holds 

Qi(P (1) h, ■ ■ ■ ,P (jB) /t) < « 9 7 (P (1) > • • • iP (B) ) < «7 

B 

l{p( 6 ) < 07 } > B'y •<=>■ ^(cry) > 7 . 

6=1 


Q( 7 ) < « 
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Thus 


P(Q(y) < a) = E(l{Q(y) < a}) = E(l{ 7 r(ay) > 7 }) < -E( 7 r(ay)) 

7 


1 _/ 1 


B 


7 \B 




b=l 


B 


< «711 = 


E^ £ “T> £ } 4 E 5> : '{C.n S<« / 0} 


1 1 

7 B 


B 


B 


E e 

6=1 


< ay 


0:7 


^ rr ' Sa ^ 7 ^#) 

aAl{Cn 5( fe ) / 0} 

~B ^ 


6=1 


m: 


(b) 


where the first inequality is a consequence of the Markov inequality and the last inequality is a 
consequence of the assumptions that P(p^ < a) = P(p^m^ < a) < a/m^ and the definition 
p(b) = l for C n = 0. 

For a random variable U taking values in [0,1], 


sup 

7e(7min,l) 


1 {U < 07 } 


7 


and if U has an uniform distribution on [0,1] 

1{U < ay}' 


E ( sup 

^7e(7min,l) 


7 


0, 

U > a 

a/U, 

Q ; 'Tmin — U <C Ql. 

1 / 7min t 

U < G^min* 

'Ot 7min 

PCX. 


1 x=a lmi 
'min" 1 ' x=0 


7 min dx + ax dx 

J a7min 

X—OL 


+ a log x 

° x=a^ min 


a + a (log a - log(a 7 min )) 

Oi 

a(l - log ——) = a(l - log 7 min ). 

« 7 min 


We apply this using as U the uniform distributed p^/m^ = p ^ for C n S^ / 0 and obtain 

l{p(b)/m( b ) < ay}' 


E ( sup 

^7G(7mind) 


-) < a(l - logymin), 
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and similarly as above 


inf Qin) < a) = e(" sup ll^cry) > 7 }') 

V 7e(7min,i) J 


'7e(7min,l) 


/ 1 1 {p^ < 07 } 

< M su p rE- “ 

V 7e(7min,l) B f^i 


7 


B 


„( 1 f ip (fc) <«7icns(^0 

E ( SU P r/L- 

V 76(7min,l) ^ ^ 


7 


B 


< 


i y, / ^ l{p^/m^ < a^/m^}l{CnS^ + 0} 


fc=1 V 7e(7min,l) 
B 


< ,i-,o g , mlo )|5:y onsW ^> 

b= 1 


m 


( 6 ) 


□ 


7.3 Additional empirical results 


p 

FWER 

# MTDs 

# STDs 

SB 

SH 

HB 

HSR 

SB 

SH 

HB 

HSR 

HB 

HSR 



“small blocks”- 

design 

with high SNR 



0 

0 

0 

0 

0 

9.87 

9.89 

9.90 

9.90 

9.86 

9.86 

0.4 

0 

0 

0 

0 

10 

10 

10 

10 

10 

10 

0.7 

0 

0 

0 

0 

10 

10 

10 

10 

10 

10 

0.8 

0 

0 

0 

0 

9.85 

9.89 

9.98 

9.98 

9.90 

9.91 

0.85 

0 

0 

0 

0 

9.26 

9.38 

9.89 

9.92 

9.39 

9.53 

0.9 

0 

0 

0 

0 

9.59 

9.65 

10 

10 

9.67 

9.79 

0.95 

0.21 

0.23 

0.21 

0.28 

8.36 

8.46 

9.82 

9.78 

8.36 

8.61 

0.99 

0.92 

0.93 

0.92 

0.95 

6.72 

6.85 

8.06 

8.04 

6.73 

6.99 



“1 

urge blocks”-' 

design 

with high SI 

MR 



0 

0 

0 

0 

0 

10 

10 

10 

10 

10 

10 

0.4 

0 

0 

0 

0 

9.98 

9.98 

10 

10 

9.99 

9.99 

0.7 

0 

0 

0 

0 

5.12 

5.35 

9.60 

9.60 

5.10 

5.12 

0.8 

0 

0 

0 

0 

9.23 

9.43 

10 

10 

9.14 

9.15 

0.85 

0 

0 

0 

0 

3.86 

4.03 

9.98 

9.98 

3.84 

3.85 

0.9 

0 

0 

0 

0 

0.06 

0.06 

7.17 

7.17 

0.06 

0.06 

0.95 

0 

0 

0 

0 

1.26 

1.29 

9.99 

9.99 

1.27 

1.28 

0.99 

0.33 

0.33 

0.99 

0.99 

3.26 

3.26 

7.92 

7.92 

3.26 

3.26 


Table 3: Results of the simulation with the “small blocks”- and “large blocks”-design with high 
SNR (SNR= 8 ) for 8 different correlations p in the design, for single variable method with Bonferroni 
(SB), single variable method with Bonferroni-Holm (SH), hierarchical method with Bonferroni (HB) 
and hierarchical method with sequential rejection induced by the inheritance procedure (HSR). 
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p 

FWER 

# MTDs 

# STDs 

SB 

SH 

HB 

HSR 

SB 

SH 

HB 

HSR 

HB 

HSR 



“small blocks”- 

design 

with high SNR 



0 

0 

0 

0 

0 

9.57 

9.69 

9.53 

9.63 

9.42 

9.53 

0.4 

0 

0 

0 

0 

8.84 

9.06 

8.65 

8.81 

8.36 

8.51 

0.7 

0 

0 

0 

0 

5.87 

6.26 

7.28 

7.60 

5.65 

6.13 

0.8 

0 

0 

0 

0 

5.53 

5.76 

6.79 

7.22 

5.33 

5.89 

0.85 

0.03 

0.04 

0.03 

0.04 

2.97 

3.08 

5.21 

5.56 

2.82 

3.14 

0.9 

0.01 

0.01 

0.01 

0.01 

3.35 

3.55 

5.49 

5.86 

3.22 

3.60 

0.95 

0.46 

0.47 

0.46 

0.48 

1.02 

1.11 

4.04 

4.07 

0.9 

0.99 

0.99 

0.55 

0.56 

0.54 

0.58 

3.62 

3.78 

6.01 

6.27 

3.42 

3.71 



U 

large blocks”- 

■design 

with 

ow Sh 

JR 



0 

0 

0 

0 

0 

8.42 

8.68 

8.38 

8.50 

7.98 

8.11 

0.4 

0 

0 

0 

0 

7.61 

8.09 

8.98 

8.98 

7.44 

7.48 

0.7 

0 

0 

0 

0 

0.67 

0.71 

5.90 

5.91 

0.59 

0.59 

0.8 

0 

0 

0 

0 

0.27 

0.27 

6.02 

6.02 

0.24 

0.24 

0.85 

0 

0 

0 

0 

0 

0 

3.38 

3.38 

0 

0 

0.9 

0 

0 

0.06 

0.06 

0.38 

0.39 

7.59 

7.60 

0.38 

0.38 

0.95 

0.03 

0.03 

0.16 

0.16 

0.45 

0.45 

8.67 

8.68 

0.44 

0.44 

0.99 

0.97 

0.97 

1.00 

1.00 

1.47 

1.48 

5.28 

5.27 

1.47 

1.48 


Table 4: Results of the simulation with the “small blocks”- and “large blocks”-design with low SNR 
(SNR=4) for 8 different correlations p in the design, for single variable method with Bonferroni 
(SB), single variable method with Bonferroni-Holm (SH), hierarchical method with Bonferroni (HB) 
and hierarchical method with sequential rejection induced by the inheritance procedure (HSR). 
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